Displaying image data based on perspective center of primary image

ABSTRACT

A method for displaying image data from a plurality of images on a display monitor, where at least some of the images are acquired from different locations, includes displaying the image data from the images on the display monitor. The image data may be displayed from a perspective center associated with the different locations from which the images are acquired. A primary image is determined based on a portion of the image data from the primary image that is displayed on the display monitor compared to portions of the image data from other images that are displayed on the display monitor. Thereafter, the image data on the display monitor is displayed from a perspective center of the primary image. A stacking order of the plurality of images is arranged so that the primary image is on top.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 61/624,296, filed Apr. 15, 2012, the entire contents of which areincorporated herein by reference in their entirety for all purposes.

FIELD OF THE INVENTION

The present disclosure relates in general to image display improvements,and more particularly, to improved methods for displaying image data andlocating points of interest in images.

BACKGROUND

Traditional surveying may involve operators working with a theodoliteand range pole. One operator generally positions the theodolite over aknown point while the other operator holds the range pole at a series ofpoints whose locations are to be determined. A target mounted on therange pole is sighted through the theodolite and accurate angularmeasurements are obtained. Accurate distance measurements may also beobtained using a measuring tape or any other distance measuring device.The locations of the points can be determined using the angular anddistance measurements in accordance with known triangulation techniques.

In modern satellite based surveying, an operator may move about with asurvey instrument that determines position information using a globalnavigation satellite system (GNSS). The satellite positioning systemmost commonly used today is the global positioning system (GPS),although others such as the global orbiting navigation system (GLONASS)are in use or under development. The operator generally moves about asite and stops at various points to record location informationdetermined using signals transmitted by satellite sources. Correctiondata may be transmitted from a reference site using a telemetry system.

Cameras are often used with modern survey instruments to associatelocation information with image information. The combined informationcan be used with known photogrammetry techniques to determinethree-dimensional coordinates of points. Such techniques often use twoor more overlapping or adjacent images taken from different stations(e.g., different locations). Cameras can be arranged to obtain imagesthat provide a continuous view of a scene. Many survey instruments, forexample, have one or more cameras configured to obtain one or moreimages that provide up to a full 360° view.

The location and image information can be processed at a site using acontroller or may be processed at an office remote from the site using acomputer system. The image information allows for additionalmeasurements to be obtained at any time as long as the additional pointsor objects are visible in the images.

As the use of location and image information becomes more widespread,techniques to utilize and view the information become increasingimportant. Thus, improved methods for displaying image data and locatingpoints of interest in images are continuously desired.

SUMMARY

Embodiments of the present invention provide improved methods fordisplaying image data and locating points of interest in displayedimages. For example, in accordance with an embodiment of the presentinvention, a method for displaying arrays of digital image data on adisplay monitor as a continuous image, where each array forms atwo-dimensional image of a different portion of a scene captured by animaging device, and where shapes of surfaces and objects in the sceneare unknown and at least some of the arrays of digital image data arecaptured from locations that are different from a location of aperspective center from which the arrays of digital image data aredisplayed, comprises determining nonlinear corrections for each of thearrays of digital image data. Each nonlinear correction may be based inpart on a location of the imaging device used to capture the array ofdigital image data relative to a location of the perspective center fromwhich the array of digital image data will be displayed on the displaymonitor. The method also includes transforming each array of digitalimage data using the corresponding nonlinear correction, where for eacharray of digital image data, the corresponding nonlinear correctionmodifies pixels in one portion of the array of digital image datadifferently than pixels in another portion of the array of digital imagedata to produce arrays of corrected image data. The method also includesdisplaying the arrays of corrected image data on the display monitor toproduce a continuous display of the scene captured by the imagingdevice, where relative sizes and positions of objects in the scene aresubstantially reproduced in the display of the scene.

In an embodiment, at least a portion of each two-dimensional image and acorresponding portion of the scene overlap with another two-dimensionalimage and a corresponding portion of the scene, and pixels in differentarrays of corrected image data that capture substantially similar lightrays from the scene are substantially overlying each other in thecontinuous display of the scene.

In another embodiment, adjacent portions of the scene are captured bydifferent arrays of digital image data to form adjacent two-dimensionalimages of the scene, and pixels in adjacent two-dimensional images thatcapture substantially adjacent light rays from the scene aresubstantially adjacent each other in the continuous display of thescene.

In another embodiment, the imaging device comprises a plurality ofimaging devices each capturing one of the arrays of digital image data,each of the imaging devices having a fixed position and orientationrelative to the other imaging devices.

In another embodiment, the imaging device comprises a plurality ofimaging devices each capturing one of the arrays of digital image data,the plurality of imaging devices coupled to a survey instrument andarranged such that a field of view of each imaging device overlaps withfields of view of other imaging devices to provide a 360° field of viewof the scene surrounding the plurality of imaging devices.

In another embodiment, the imaging device comprises a digital cameraconfigured to rotate about a support mechanism to provide the arrays ofdigital image data, each array of digital image data having a fixedpositional relationship with adjacent arrays of digital image data.

the arrays of digital image data form overlapping two-dimensional imagesof the scene to provide a 360° panoramic view of the scene.

each nonlinear correction is based in part on a distance between a lensof the imaging device used to capture the array of digital image dataand a concave-shaped surface.

each nonlinear correction is based in part on a distance between a lensof the imaging device used to capture the array of digital image dataand a flat portion of a surface having a plurality of flat portionsarranged in a concave shape.

In another embodiment, the imaging device comprises a plurality ofdigital cameras.

In another embodiment, each of the nonlinear corrections change arelative size of the pixels across the corresponding array of digitalimage data.

In yet another embodiment, the method also includes providingcoordinates of a point in the scene, the coordinates provided in areference frame associated with the imaging device, and superimposing acomputer-rendered graphic on the display of the scene at projectedcoordinates of the point such that the computer-rendered graphicoverlies the point in the display of the scene.

In accordance with another embodiment of the present invention, a methodfor displaying image data from a plurality of images on a displaymonitor, where at least some of the plurality of images are acquiredfrom different locations, includes displaying the image data from theplurality of images on the display monitor. The image data from theplurality of images may be displayed from a perspective centerassociated with the different locations from which the plurality ofimages are acquired. The method also includes determining a primaryimage of the plurality of images. The primary image may be determinedbased on a portion of the image data from the primary image that isdisplayed on the display monitor compared to portions of the image datafrom other images that are displayed on the display monitor. The methodalso includes displaying the image data from the plurality of images onthe display monitor. The image data from the plurality of images may bedisplayed from a perspective center of the primary image. The methodalso includes arranging a stacking order of the plurality of images sothat the primary image is on top.

In some embodiments, the plurality of images capture overlappingportions of a scene. In other embodiments, the plurality of imagescapture adjacent portions of a scene.

In another embodiment, the perspective center associated with thedifferent positions from which the plurality of images are acquired islocated substantially at an average position of the different positions.

In another embodiment, the primary image is also determined based on aposition of the primary image on the display monitor relative topositions of the other images displayed on the display monitor.

In yet another embodiment, the method also includes forming a linearound the primary image as an indication of the primary image.

In accordance with another embodiment of the present invention, a methodfor identifying one or more pixels of image data associated with a pointin an image using a display monitor displaying the image includesreceiving an indication of a portion of the image as displayed on thedisplay monitor that surrounds the point, and identifying pixels ofimage data as displayed on the display monitor that are locatedapproximately at edges of the portion of the image. The method alsoincludes performing one or more inverse transformations to transform thepixels of image data as displayed on the display monitor to pixels ofimage data as captured by an image sensor, and identifying a portion ofimage data as captured by the image sensor that corresponds to a portionof the image that surrounds the point, the portion of the image dataidentified based on the pixels of image data as captured by the imagesensor. The method also includes displaying the portion of the imagedata as captured by the image sensor on the display monitor without datatransformation, and receiving an indication of the one or more pixels ofimage data associated with the point in the image.

In an embodiment, the portion of the image on the display monitor thatsurrounds the point is polygon shaped, and the pixels of the displaymonitor that are located approximately at the edges of the portion ofthe image are located approximately at corners of the polygon.

In another embodiment, the method also includes magnifying the portionof the image data as captured by the image sensor. Displaying theportion of the image data as captured by the image sensor may comprisedisplaying the magnified portion of the image data.

In accordance with yet another embodiment of the present invention, amethod for locating a point of interest in a plurality of imagesincludes identifying the point of interest in a first view of the scene.The point of interest may be identified by selecting a point on thedisplay monitor that corresponds to the point of interest as displayedin the first view. The method also includes identifying pixels of imagedata that correspond to the selected point on the display monitor. Thepixels of image data may be part of one or more arrays of image datathat form the first view of the scene. The method also includesdetermining a line in each of the views. The line may extend along a raythat passes through the point of interest in the scene to pixels of animage sensor corresponding to the pixels of image data in the first viewof the scene. The method also includes displaying the line in each ofthe views in which the line is visible, and providing a slidableindicator in each of the views that is movable along the line. Theslidable indicator can be moved by user input, and any movement of theslidable indicator along the line may be visible in each of the otherviews and updates the scene displayed in each of the other viewsaccordingly. The method also includes moving the slidable indicatoralong the line until the slidable indicator is proximate the point ofinterest in each of the other views.

In an embodiment, the first view of the scene comprises one or moreimages each displayed from a common perspective center.

In another embodiment, each of the views comprise more than one image,and for each view, the corresponding images are obtained from adifferent station and displayed on a window in the display monitor froma common perspective center.

In another embodiment, the different stations are at different locationswithin the scene.

In another embodiment, the point of interest is identified in the firstview by selecting a point on a display monitor using an input device

In another embodiment, the slidable indicator is a slide bar.

In another embodiment, as the slidable indicator is moved, the view ineach of the other views is oriented to provide a side view of theslidable indicator moving along the line.

In another embodiment, a length of the line is determined based at leastin part on a distance between the different stations from which each ofthe views are obtained, and as the slidable indicator is moved along theline, a distance from the image sensor to the slidable indicator isdetermined.

In yet another embodiment, a zoom level of each of the other windows isadjusted as the slidable indicator is moved along the line.

Numerous benefits are provided by embodiments of the present inventionover conventional techniques. In some embodiments, for example, imagedata captured from different locations can be displayed on a displaymonitor (e.g., a controller screen, computer monitor, or the like) sothat it closely resembles a real-world scene it represents. This meansthat the displayed image data has a high metric value (or that relativesizes and positions of objects in the scene are substantiallyreproduced). This can be done without knowledge of the shapes ofsurfaces and objects in the real-world scene. This allows the image datato be used for accurate survey and photogrammetry measurements. Thisalso ensures that the image data will closely align with positions ofcorresponding points or objects that are superimposed onto the imagedata. For example, positions of several objects around a site may bemeasured using conventional surveying techniques. Images of the site mayalso be obtained. When viewing the image data on a display monitor, itmay be useful to superimpose graphics (e.g., points, signs, coordinates,or the like) over the image data to identify the measured objects. Thedisplayed image data corresponding to the objects will align with thesuperimposed graphics if the image data has a high metric value or isdisplayed in accordance with embodiments of the invention.

In other embodiments, points of interest in displayed images can bequickly located and accurately identified. For example, in an embodimenta reference graphic that passes through a point of interest may besuperimposed over a displayed image captured from a first station. Thereference graphic may also be superimposed over other displayed imagescaptured from other stations. The reference graphic can be followed tothe point of interest in the other displayed images to quickly locatethe point of interest in those images. In another embodiment, pixels ofimage data without transformation can be used to select a point ofinterest in a displayed image. This allows a location of the point to beaccurately determined.

Depending on the embodiment, one or more of the benefits may exist.These and other benefits are described throughout the specification.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1B are simplified diagrams of survey instruments that may beused in accordance with some embodiments of the invention;

FIGS. 2A-2B are simplified diagrams showing various aspects of anexemplary image sensor;

FIG. 3 is a simplified plan view of an imaging device that may be usedin accordance with some embodiments of the invention;

FIGS. 4A-4B and FIGS. 5A-5B are simplified diagrams illustrating imagedata projected onto surfaces in accordance with some embodiments of theinvention;

FIG. 6 is a flowchart illustrating a method for displaying arrays ofdigital image data on a display monitor in accordance with an embodimentof the invention;

FIG. 7 is a simplified diagram illustrating some of the geometricalconcepts that are involved in determining nonlinear corrections inaccordance with an embodiment of the invention;

FIGS. 8A-8B are simplified plan views showing a series of overlappingimage data and portions of the image data that are displayed on adisplay monitor in accordance with some embodiments of the invention;

FIG. 9 is a flowchart illustrating a method for displaying image datafrom a plurality of images on a display monitor in accordance with anembodiment of the invention;

FIG. 10 is a simplified diagram showing pixels of a display monitor andcorresponding pixels of image data on an image sensor in accordance withan embodiment of the invention;

FIG. 11 is a flowchart illustrating a method for identifying one or morepixels of image data associated with a point in an image using a displaymonitor displaying the image in accordance with an embodiment of theinvention;

FIGS. 12A-12B are simplified diagrams showing imaging devices atdifferent stations around a site and a point of interest that iscaptured in images from each of the stations in accordance with anembodiment of the invention; and

FIG. 13 is a flowchart illustrating a method for locating a point ofinterest in a plurality of images in accordance with an embodiment ofthe invention.

DETAILED DESCRIPTION

Embodiments of the present invention provide improved methods fordisplaying image data and locating points of interest in displayedimages. Some embodiments enable image data captured from differentlocations to be displayed on a monitor so that it closely resembles areal-world scene it represents. This can be done using a nonlinearcorrection that is based on a location of an imaging device used tocapture an array of image data relative to a location of a perspectivecenter from which the array of image data will be displayed. Thenonlinear correction can be determined without knowledge of the shapesof surfaces and objects in the real-world scene. Other embodiments allowthe image information to be used more easily so that points or objectsin images can be quickly located and accurately identified. In anembodiment, this can be done using a reference graphic that issuperimposed over a displayed image and that passes through a point ofinterest. The reference graphic can be followed to the point of interestin other displayed images that are captured from different stations. Inanother embodiment, pixels of image data without transformation can beused to select a point of interest in a displayed image. These and otherembodiments are described more fully below.

FIGS. 1A-1B are simplified diagrams of survey instruments that may beused in accordance with some embodiments of the invention. It should beappreciated that these survey instruments are provided merely asexamples and other instruments having different configurations maysimilarly be used in accordance with the various embodiments.

FIG. 1A shows a survey instrument 100 that includes a GNSS receiver 102and an imaging device 104 coupled to a support pole 106. The surveyinstrument 100 may also include other components such as one or moretilt sensors, distance measurement systems, or stabilizers. The supportpole 106 has a longitudinal axis and may be designed so that a bottomtip of the support pole 106 can be accurately placed at a point ofinterest while measuring a position of the point of interest and/orcapturing an image at the point of interest. The support pole 106 may beconstructed of any suitable material such as wood, metal, plastic,composite, or the like and any combinations thereof. The support pole106 may be monolighic or may comprise two or more separate sections thatcan be fixed or coupled together during manufacture or operation. Thesupport pole 106 may include a compartment to store batteries thatsupply power to the GNSS receiver 102, the imaging device 104, and/orany other components that may be coupled to the support pole 106 inalternative configurations.

The GNSS receiver 102 may be permanently fixed to or removably coupledwith the support pole 106. In certain embodiments, the GNSS receiver 102may be affixed to a top end of the support pole 106 or otherwisedisposed on or near a top portion of the support pole 106. The GNSSreceiver 102 may include a receiver, an antenna, and/or other equipmentnecessary to receive signals and to determine position information. TheGNSS receiver 102 may employ any of a variety of known satellitenavigation techniques. For example, the GNSS receiver 102 may beconfigured to employ real-time kinematic (RTK) techniques in conjunctionwith reception of correction signals. The GNSS receiver 102 may also beused to determine orientation information. As used herein, the term“orientation” means any disposition of the survey instrument 100relative to a reference (such as magnetic North or True North). As anexample, the GNSS receiver 102 may be rotated in an arc, and multiplesets of location information obtained during rotation can be used todetermine orientation of the GNSS receiver 102. Alternatively, the GNSSreceiver 102 may include a compass and/or a compass or other orientationdevice may be separately coupled to the support pole 106.

The GNSS receiver 102 may also include a power supply (e.g., arechargeable battery, and input from an external power source, or thelike), non-transitory computer-readable storage media (e.g., magneticstorage, optical storage, solid-state storage, or the like), and acommunication interface. The communication interface may comprisehardware and/or software necessary for communication with othercomponents of the survey instrument 100 and/or external devices such asa controller 108. The communication interface may include a UHF radio, acellular radio, a Bluetooth transceiver, a Wi-Fi transceiver, a wiredcommunication interface, and/or the like.

The GNSS receiver 102 and the imaging device 104 may be configured tocommunicate with the controller 108. The controller 108 may include anydevice or combination of devices that function to control and/or receiveinput from components of the survey instrument 100. The controller 108may include a communication interface similar to that of the GNSSreceiver 102 described above. The communication interface of thecontroller 108 may be configured to support long-range communications(e.g., cellular or WWAN) even if the GNSS receiver 102 and imagingdevice 104 are not. Further, the communication interface of thecontroller 108 may be configured to allow communication with otherdevices to exchange information.

The controller 108 may include a special-purpose computing device or ageneral-purpose computing system. The controller 108 may be configuredto process input received from the survey instrument 100, sendinstructions to the survey instrument 100, and to perform methods inaccordance with embodiments of the present invention. The controller 108may include one or more input devices (e.g., buttons, keypad, keyboard,touchscreen, mouse, and the like) and/or one or more output devices(e.g., display monitor, printer, and the like). The controller 108 willtypically include one or more processors and non-transitorycomputer-readable storage media (e.g., magnetic storage, opticalstorage, solid-state storage, or the like).

In some embodiments, the controller 108 provides a user interface to thesurvey instrument 100 via the input and output devices. The userinterface allows a user to a control operation of the survey instrument100. A variety of user interfaces may be provided including graphicaluser interfaces that display one or more screens for providing output toa user and/or for receiving input from a user.

In some embodiments the user interface and/or functionality of thecontroller 108 may be provided remotely. Merely by way of example, thecontroller 108 and/or the survey instrument 100 may be configured tocommunicate with a separate computer (e.g., an office computer orcomputer system). In this scenario, at least a portion of the userinterface and functionality provided by the controller 108 may beprovided by the remote computer. The computer may serve to communicateinstructions to the controller 108 and/or to receive data therefrom.Thus, it should be appreciated that the methods described herein couldbe implemented locally (e.g., in the field) using the controller 108,remotely (e.g., in an office) using a separate computer, or somecombination thereof. In other embodiments, the user interface andfunctionality may be provided through a web site that may be displayedusing a web browser running on the controller 108 and/or the computer.

A number of different configurations of the controller 108 are possible.In one embodiment, for example, the controller 108 may be permanently(or relatively permanently) mounted on the support pole 106. In anotherembodiment, the controller 108 may be formed integrally with the supportpole 106. In yet other embodiments, the controller 108 may be removablymounted to the support pole 106 or physically separate from the supportpole 106.

The imaging device 104 may include any device (or set of devices) thatis capable of capturing optical images. The imaging device 104 mayinclude one or more digital cameras, digital video cameras, or the like.The imaging device 104 typically includes a lens that is used to providelight rays to a sensor array (e.g., and a CCD or CMOS optical sensor).The light rays are reflected off objects, surfaces, and points and thatexist in a scene surrounding the imaging device 104 (or a portion of thescene within a field of view of the imaging device 104). The light rayspass through the lens and impinge on pixels (or discrete elements) ofthe array to form electrical signals that collectively provide imagedata (or an array of image data). This can be illustrated with referenceto FIG. 2A, where light rays 202 a, 202 b pass through a lens 206 andimpinge on pixels 204 a, 204 b of sensor array 208. As would beappreciated by one of ordinary skill in the art, a typical imagingdevice may include a sensor array with millions of pixels. Further, inreality the light rays are not discrete elements but instead are part ofa continuum of rays emitted from objects in the scene within the fieldof view of the imaging device.

As illustrated in FIG. 2B, image data collected by pixels 204 of animage sensor 208 may be stored in memory 214 and/or provided to aprocessor 216 for processing. The memory 214 may be any type ofconventional data storage (e.g., magnetic storage, optical storage,solid-state storage, or the like) configured to store the image data.The memory 214 may also store instructions that are used by theprocessor 216 to process the image data. It should be appreciated thatthe memory 214 and/or the processor 216 may be located locally withinthe imaging device 104 or remote from the imaging device 104 (e.g.,within the controller 108 and/or within a remote computer).

The imaging device 104 may include a communication interface similar tothat of the GNSS receiver 102 described above. Image data may becommunicated to the controller 108 or another computer using thecommunication interface. The imaging device 104 may include anynecessary hardware and software to store the captured image data (e.g.,arrays of digital image data), process the captured image data, and/ortransmit the captured image data to another device (such as thecontroller 108). The imaging device 104 may or may not have a separatedisplay for viewing the image data.

In some embodiments, the imaging device 104 may include a plurality ofimaging devices arranged to collectively capture a series of imagesproviding a panoramic view of a scene surrounding the survey instrument100. In some embodiments the panoramic view may include a full 360°view. In the example shown in FIG. 1A, the imaging device 104 includes anumber of imaging devices arranged in a cluster about the support pole106. The imaging devices may be arranged in a fixed relationship to eachother (e.g., with a fixed angle between them) and configured to capturea continuous view of the scene almost simultaneously. This is incontrast to the example shown in FIG. 1B, where the imaging device 114includes a single imaging device (or fewer imaging devices). In thisembodiment, the imaging device 114 may be configured to rotate about thesupport pole 106 either continuously or in increments to provide aseries of images capturing a continuous view of the scene. The GNSSreceiver 102 and support pole 106 shown in this example may beconfigured in a manner similar to their corresponding components shownin FIG. 1A and described above.

FIG. 3 is a simplified plan view of an imaging device that may be usedin accordance with some embodiments of the invention. The imaging devicecan be used to provide a series of images that can be displayed togetherto provide a continuous view of a scene. In this example, an imagingdevice 304 includes eight separate imaging devices 310 a, 310 b, 310 c,310 d, 310 e, 310 f, 310 g, 310 h arranged about a circle and havingfields of view 312 a, 312 b, 312 c, 312 d, 312 e, 312 f, 312 g, 312 h.The fields of view overlap providing overlapping portions 314 a, 314 b,314 c, 314 d, 314 e, 314 f, 314 g, 314 h. It should be appreciated thatthis embodiment is provided merely as an example, and more or fewerimaging devices providing more or fewer fields of view may be used inaccordance with embodiments of the invention. In an alternativeembodiment, for example, the imaging device 304 could include fewerimaging devices that are configured to obtain images at positions of theimaging devices 310 a, 310 b, 310 c, 310 d, 310 e, 310 f, 310 g, 310 hshown in this example. Other embodiments may utilize an imaging devicethat provides a view encompassing less than the 360° view shown in thisexample. Further, in some embodiments, the imaging device 304 couldinclude imaging devices arranged about both horizontal and vertical axisto expand coverage of the fields of view up to and including a completespherical view.

Displaying Image Data

FIGS. 4A-4B are simplified diagrams showing image data projected ontosurfaces in accordance with some embodiments of the invention. Each ofthese figures show image data 424 a, 424 b, 424 c, 424 d, 424 e, 424 f,424 g, 424 h projected onto a surface 428 from a location 432 a, 432 b,432 c, 432 d, 432 e, 432 f, 432 g, 432 h. The locations 432 a, 432 b,432 c, 432 d, 432 e, 432 f, 432 g, 432 h are at substantially the samelocations from which the image data 424 a, 424 b, 424 c, 424 d, 424 e,424 f, 424 g, 424 h was obtained (the locations of the individualimaging devices). The image data 424 a, 424 b, 424 c, 424 d, 424 e, 424f, 424 g, 424 h between adjacent images may overlap depending on adistance from the locations 432 a, 432 b, 432 c, 432 d, 432 e, 432 f,432 g, 432 h to the surface 428 to provide overlapping regions 426 a,426 b, 426 c, 426 d, 426 e, 426 f, 426 g, 426 h.

FIGS. 4A-4B also show a perspective center 422 near a center of imagingdevice 404. While the image data 424 a, 424 b, 424 c, 424 d, 424 e, 424f, 424 g, 424 h is captured from different locations 432 a, 432 b, 432c, 432 d, 432 e, 432 f, 432 g, 432 h, it is displayed on a displaymonitor from a perspective of a single location. This location istypically the perspective center 422 but, as explained below with regardto FIGS. 8A-8B and FIGS. 9A-9B, the image data may be displayed fromother locations in certain situations. A location of the perspectivecenter 422 is generally selected at an average of the locations 432 a,432 b, 432 c, 432 d, 432 e, 432 f, 432 g, 432 h from which the imagingdata is captured and at a center of curvature of surface 428 andsurfaces 430 a, 430 b, 430 c, 430 d, 430 e, 430 f, 430 g, 430 h.

Note that the surfaces 428 and 430 a, 430 b, 430 c, 430 d, 430 e, 430 f,430 g, 430 h shown in FIGS. 4A-4B are part of a plan view and thesurfaces may extend in a third dimension to form spheres or cylinders.Imaging devices may also be arranged to capture image data about ahorizontal axis to provide additional image data of the scene aboveand/or below the horizontal plane shown in these examples. It should beappreciated that the methods described below are equally applicable toimage data captured of any view of a surrounding scene (including acomplete spherical view).

It should be appreciated that the surface 428 illustrated in FIG. 4A andthe surfaces 430 a, 430 b, 430 c, 430 d, 430 e, 430 f, 430 g, 430 hillustrated in FIG. 4B do not represent actual physical surfaces.Instead, these surfaces represent mathematical constructs upon which itis presumed the image data will be projected for purposes of determiningnonlinear corrections for the image data (or parameters for nonlineartransformation of the image data). Further, the image data is notactually projected onto the surface 428 or the surfaces 430 a, 430 b,430 c, 430 d, 430 e, 430 f, 430 g, 430 h when determining the nonlinearcorrections or during transformation of the image data, but the imagedata is mathematically transformed as if it were to be projected ontothese surfaces. This is in contrast to displaying the image data on adisplay monitor where transformed (or corrected) image data is displayedfor viewing by a user.

The nonlinear corrections are parameters that are used to transform theimage data. The image data is transformed from a two-dimensional arrayof image data collected by an image sensor to an array of correctedimage data that can be provided to a graphics system for display on adisplay monitor. The graphics system may display the corrected imagedata in a perspective manner. The corrected image data can be used toprovide images on a display monitor that have a high metric value. Thismeans that relative sizes and positions of objects in a scene aresubstantially reproduced in the displayed image. Further, because anglesbetween adjacent arrays of image data are known (or angles between theimaging devices use to obtain the image data are known), the arrays ofadjacent image data (such as image data 424 a and image data 424 b shownin FIG. 4A) can be displayed as part of a continuous scene without usinga feature matching algorithm to stitch the images (or distort theimages) and without having a knowledge of the shapes of surfaces and/orobjects in the scene.

The nonlinear corrections may be based on a difference between alocation of an imaging device that is used to capture the image data anda location of a perspective center from which the image data will bedisplayed on a display monitor. In the examples shown in FIGS. 4A-4B,this difference can be seen as the difference between the locations 432a, 432 b, 432 c, 432 d, 432 e, 432 f, 432 g, 432 h and the location ofthe perspective center 422. The nonlinear corrections may also be basedon a shape of a surface onto which the image data is projected (e.g.,the surface 428 or one of the surfaces 430 a, 430 b, 430 c, 430 d, 430e, 430 f, 430 g, 430 h) and a distance from the surface to an imagingdevice used to capture the image data (or to a lens of the imagingdevice). The nonlinear corrections may also be based on other parameterssuch as internal calibration of the imaging device and/or distortionsintroduced by a lens of the imaging device.

A shape of the surface onto which the data is projected (e.g., thesurface 428 or one of the surfaces 430 a, 430 b, 430 c, 430 d, 430 e,430 f, 430 g, 430 h) may be determined based on the lens of the imagingdevice used to capture the image data. For example, image data capturedby an imaging device using an conventional F-theta lens may be projectedonto a concave-shaped surface such as the surface 428 illustrated inFIG. 4A. Image data captured by an imaging device using a conventionalrectilinear lens may be projected onto a flat surface such as one of thesurfaces 430 a, 430 b, 430 c, 430 d, 430 e, 430 f, 430 g, 430 hillustrated in FIG. 4B. Note that the flat surfaces illustrated in FIG.4B are arranged to form a concave-shaped surface so that a distance fromthe perspective center 422 b to each of the surfaces 430 a, 430 b, 430c, 430 d, 430 e, 430 f, 430 g, 430 h is the same. Other surface shapesmay be used for other lens types.

FIGS. 5A-5B are similar to FIGS. 4A-4B, except that image data 524 a,524 b, 524 c, 524 d, 524 d, 524 e, 524 f, 524 g, 524 h does not overlap.Because the embodiments described above do not rely on conventionalfeature matching algorithms, overlap between adjacent image data is notrequired to display the image data on a display monitor as a continuousimage of a scene having a high metric value.

FIG. 6 is a flowchart illustrating a method for displaying arrays ofdigital image data on a display monitor in accordance with an embodimentof the invention. The method includes determining nonlinear correctionsfor each of the arrays of digital image data, each nonlinear correctionbased in part on a location of the imaging device used to capture thearray of digital image data relative to a location of the perspectivecenter from which the array of digital image data will be displayed onthe display monitor (602). The method also includes transforming eacharray of digital image data using the corresponding nonlinearcorrection, where for each array of digital image data, thecorresponding nonlinear correction modifies pixels in one portion of thearray of digital image data differently than pixels in another portionof the array of digital image data to produce arrays of corrected imagedata (604). The method also includes displaying the arrays of correctedimage data on the display monitor to produce a continuous display of thescene captured by the imaging device, wherein relative sizes andpositions of objects in the scene are substantially reproduced in thedisplay of the scene (606).

It should be appreciated that the specific steps illustrated in FIG. 6provide a particular method for displaying arrays of digital image dataon a display monitor according to an embodiment of the presentinvention. Other sequences of steps may also be performed according toalternative embodiments. For example, alternative embodiments mayperform the steps outlined above in a different order. Moreover, theindividual steps illustrated in FIG. 6 may include multiple sub-stepsthat may be performed in various sequences. Furthermore, additionalsteps may be added or removed depending on the particular application.

Example of Determining Nonlinear Corrections for a Concave-ShapedSurface

The following example outline one method that may be used to determinenonlinear corrections for a concave-shaped (or spherical) surface. Thesurface referenced here is a projection surface (e.g., the surface 428shown in FIG. 4A). Equations are derived to express an intersection ofrays with the surface.

Note that in this example all object space entities are considered to beexpressed in an arbitrary coordinate frame. This will typically be aworld coordinate frame (e.g. a survey coordinate system) or perhaps aninstrument coordinate frame.

A ray in space may be represented by a vector start point, s, and a unitmagnitude direction vector, r. A point on the ray, x, at a distance λfrom the start point may be expressed by the vector equation:x=s+λr  Equation (1)

If 0<λ, the point x is in front of the starting point s—i.e. it isvisible to an imaging device that has an entrance pupil at location s.If λ<0, the point x is behind the imaging device and is not visible inan image. In general, the direction vector, r, is assumed to have unitmagnitude:r ²=1  Equation (2)so that the magnitude of the scalar, λ, represents the distance betweens and x.

Consider the i-th imaging device within a survey instrument. Each pixelof an image sensor in this imaging device is associated with a geometricray in space. Let m_(j) represent the j-th pixel of interest (e.g. animage corner, a selected image location, or the like). Utilizing acombination of imaging device interior and exterior calibrationparameters, this pixel location may be converted into a photogrammetricray using the equation:m _(ij) →s _(i)+λ_(ij) r _(ij)  Equation (3)wherem_(ij) is the image sensor location (row, column) of the j-th point ofinterest measured in an image from the i-th imaging device;→ represents an application of optical and instrument calibrationparameters to associate pixel location with geometric rays in space;s_(i) is the location of the entrance pupil of the imaging deviceexpressed in the coordinate frame of interest;r_(ij) is the unit magnitude direction of the ray; andλ_(ij) is an arbitrary distance along the ray.

Note that for a single measurement from a single image, the value of thedistance parameter, λ_(ij), is unknown and must be recovered from someother means (e.g. from another image captured from a different station,by intersection with some externally available or assumed surface, oranother means).

A sphere may be represented by its vector center, c, and scalar radius,ρ. If a point, x, lies on a surface of the sphere, it satisfies theimplicit equation:(x−c)²−ρ²=0  Equation (4)

Intersection of a ray with a sphere is illustrated in FIG. 7. Locationsat which the ray intersects the sphere can be computed by insertingEquation (1) into Equation (4). For convenience, the sphere equation canbe expanded to obtain:

$\begin{matrix}\begin{matrix}{0 = {\left( {x - c} \right)^{2} - \rho^{2}}} \\{= {x^{2} - {xc} - {cx} + c^{2} - \rho^{2}}} \\{= {x^{2} - {2{x \cdot c}} + c^{2} - \rho^{2}}}\end{matrix} & {{Equation}\mspace{14mu}(5)}\end{matrix}$Then squaring the ray equation gives:

$\begin{matrix}\begin{matrix}{x^{2} = \left( {s - {\lambda\; r}} \right)^{2}} \\{= {s^{2} + {\lambda\left( {{sr} + {rs}} \right)} + \lambda^{2}}} \\{= {s^{2} + {2{\lambda\left( {s \cdot r} \right)}} + \lambda^{2}}}\end{matrix} & {{Equation}\mspace{14mu}(6)}\end{matrix}$

Note that the last term is obtained from the condition that r²=1 suchthat λ²r²=λ². Substitution of the squared ray equation and the originalinto the expanded sphere equation yields:

$\begin{matrix}\begin{matrix}{0 = {x^{2} - {xc} - {cx} + c^{2} - \rho^{2}}} \\{= {s^{2} + {2\;{\lambda\left( {s \cdot r} \right)}} + \lambda^{2} - {2{\left( {s + {\lambda\; r}} \right) \cdot c}} + c^{2} - \rho^{2}}}\end{matrix} & {{Equation}\mspace{14mu}(7)}\end{matrix}$which may be rearranged as a classic quadratic equation:

$\begin{matrix}\begin{matrix}{0 = {\lambda^{2} + {2\;{\lambda\left( {{s \cdot r} - {r \cdot c}} \right)}} + s^{2} - {2{s \cdot c}} + c^{2} - \rho^{2}}} \\{= {\lambda^{2} + {2\;{\lambda\left\lbrack {\left( {s - c} \right) \cdot r} \right\rbrack}} + \left\lbrack {\left( {s - c} \right)^{2} - \rho^{2}} \right\rbrack}}\end{matrix} & {{Equation}\mspace{14mu}(8)}\end{matrix}$

For convenience, the following substitutions may be made:β=(s−c)·r  Equation (9)γ=(s−c)²−ρ²  Equation (10)such that the quadratic equation may be expressed as:λ²+2βλ+γ=0  Equation (11)which may be solved for the two roots of λ asλ_(±)=−β±(β²−γ)^(1/2)  Equation (12)

Corresponding vector point locations, x_(±), may be determined as thesolutions at distances λ_(±) along the ray as shown by:x _(±) =s+λ _(±) r  Equation (13)

Note that the smallest positive root is the physically meaningfulsolution—i.e. is the point on the spherical surface that is visible inthe image. If both λ_(±) are positive this is the near face of thesphere. If both are negative the entire sphere is behind the imagingdevice. If the roots are identical and positive (or negative), then theray is tangent to the sphere in front of (or behind) the imaging device.If the roots for λ are imaginary, then the ray does not intersect thesphere.

As an example, consider the task of loading image textures into agraphics library. In this case, it is desirable to define a mesh ofpoints that cover the image where each mesh element consists of a numberof vertices (generally either 3 or 4). These detector locations are usedto define corners of an image texture polygon—i.e. define which pixelsare extracted from the image data file. The graphics library then“paints” this image texture polygon onto a “world polygon” that isexpressed in the 3D graphics world frame. For texturing onto a sphericalsurface, the world polygon can be computed from the image detector meshelement corners as follows.

-   -   1. Define a sphere with center, c, near the center of the        imaging device (e.g. use the instrument origin). Set the sphere        radius, ρ, to a desired viewing distance.    -   2. Define a mesh (e.g. grid) of locations covering the detector        for each imaging device.    -   3. For each mesh vertex, convert its pixel location into a ray        including start point, s and direction, r.    -   4. Use c, ρ, s, and r to compute β and γ using Equation (10).    -   5. Compute the two roots of λ using Equation (12). If the roots        are real (i.e. if 0<((β²−γ)), consider only roots with positive        values and select the smallest one.    -   6. Compute a 3D world location, x, using the λ solution value        (if one is qualifies) in Equation (1).

Note that all the world solution points, x, lie on a sphere of radius ρ.As ρ is changed, either the size of pixels on the sphere must change, orthe number of pixels on the sphere must change. For some graphicapplications and/or hardware, this may cause performance degradation.

An option is to renormalize the radius of the sphere before providingdata to the graphics application. For example, if the texturingperformance is reasonable for a sphere of radius ρ₀, then renormalizeall world coordinates computed in step 6 above—i.e. via:

$\begin{matrix}{x^{\prime} = {\frac{\rho\; 0}{\rho}x}} & {{Equation}\mspace{14mu}(14)}\end{matrix}$where, ρ₀ is the radius of good performance, and ρ is the forwardprojection radius used to compute the value of λ and hence x.

Note that for a sphere with a center near a middle of an imaging device,the separation of the ray start points and s the sphere center, (s−c) isgenerally significantly smaller in magnitude (order of 0.06 m) than thesphere radius, ρ (often order of 1-10 m)—i.e. (s−c)<<ρ. In this case:(β/γ)=[(s−c)·r]/[(s−c)²−ρ²]≅[(s−c)·r]/−ρ ²  Equation (15)is a relatively small quantity compared to one (i.e. β/γ<<1). Under thiscondition, an approximate relationship between γ and ρ may be expressedas:γ≅−ρ²  Equation (16)such that the radical may be expressed approximately as:(β²−γ)^(1/2)≅(β²+ρ²)^(1/2)  Equation (17)

Note that if |s−c|<<ρ, also β<<ρ and β²<<ρ². The radical can be writtenas:(β²+ρ²)^(1/2)=ρ[(β²/ρ²)+1]^(1/2)  Equation (18)which may be expanded using a series approximation. For small ε<<1, theTaylor series for the square root is

( 1 ± ) 1 / 2 ≃ 1 ± 1 / 2 ⁢ ⁢ ∓ 1 / 8 ⁢ 2 ± …Applying this to the radical and retaining terms of first order yields:ρ[1+(β²/ρ²)]^(1/2)≅ρ[1+½(β²/ρ²)]  Equation (19)

The solution represented by Equation (12) may therefore be approximatedas:

$\begin{matrix}\begin{matrix}{\lambda_{\pm} = {{- \beta} \pm \left( {\beta^{2} - \gamma} \right)^{1/2}}} \\{\simeq {{\pm \rho} - \beta}} \\{\simeq {{\pm \rho} - {\left( {s - c} \right) \cdot r}}}\end{matrix} & {{Equation}\mspace{14mu}(20)}\end{matrix}$

Determining a Primary Image

FIGS. 8A-8B are simplified plan views showing a series of overlappingimage data and portions of the image data that are displayed on adisplay monitor in accordance with some embodiments of the invention.These figures combine some of the features shown in FIG. 3 and FIGS.4A-4B that were described above. For example, FIGS. 8A-8B are similar toFIG. 3 in that they each show an imaging device 804 that includes anumber of imaging devices 810 having fields of view 812 with overlappingportions 814. Note that each of the individual imaging devices 810,fields of view 812, and overlapping portions 814 are not separatelyidentified in FIGS. 8A-8B to avoid excessive clutter. FIG. 8A is similarto FIG. 4A in that it shows a perspective center 822 a from which imagedata is projected onto a viewing surface 828 for purposes of determiningnonlinear corrections. Portions of the image data 824 a, 824 b, 824 coverlap shaded regions 830 a, 830 b. The shaded regions 830 a, 830 brepresent image data that forms a continuous display of a scenedisplayed on a display monitor or within a window on a display screen.

These figures show a change in a perspective center from which imagedata is displayed on a display monitor. In the example shown in FIG. 8A,portions of the image data 824 a, 824 b, 824 c that overlap with theshaded region 830 a are displayed on the display monitor from aperspective center 822 a located approximately a center of the imagingdevice 804. In the example shown in FIG. 8B, portions of the image data824 a, 824 b, 824 c that overlap with the shaded region 830 b aredisplayed on the display monitor from a perspective center 832 b locatedapproximately at a location of an imaging device that was used tocapture the image data 824 b. Thus, in going from FIG. 8A to FIG. 8B,the perspective centers from which image data is displayed on a displaymonitor shifts from the perspective center 822 a to the perspectivecenter 832 b. Similarly, the surface 828 in FIG. 8B shifts and iscentered around perspective center 832 b.

Shifting perspective centers, as illustrated in these figures, may beused in a number of scenarios according to various embodiments of theinvention. In one embodiment, the perspective center may be shifted froman average location of the imaging devices to a location of one of theimaging devices as a user zooms in on a particular portion of the imagedata. As an example, image data may be displayed from the perspectivecenter 822 a shown in FIG. 8A when the image data displayed on thedisplay monitor is zoomed out and/or when the image data displayed onthe display monitor overlaps with significant portions of the image data824 a, 824 b, 824 c (note that the shaded region 830 a overlaps with allof the image data 824 b and a large portion of the image data 824 c).Image data may be displayed from the perspective center 832 b shown inFIG. 8B when the image data on the display monitor is zoomed in and/orwhen the image data displayed on the display monitor overlaps primarilywith one only one of the image data 824 a, 824 b, 824 c (note that theshaded region 830 b overlaps primarily with the image data 824 b). Inthis case the image data to which the perspective center shifts may beconsidered the primary image and a stacking order may be arranged sothat the primary image is on top. The primary image typically has thehighest metric value since the perspective center from which it isdisplayed is substantially the same as the location from which the imagedata was acquired.

In some embodiments the primary image may change as a user pans aroundto different portions of a scene. In other embodiments, the primaryimage may change depending on which image data is displayed in a centerof the display monitor (generally indicating an area interest to auser). In yet other embodiments, the primary image may be determinedbased on some combination of the above factors.

FIG. 9 is a flowchart illustrating a method for displaying image datafrom a plurality of images on a display monitor in accordance with anembodiment of the invention. The method includes displaying the imagedata from the plurality of images on the display monitor, the image datafrom the plurality of images displayed from a perspective centerassociated with the different locations from which the plurality ofimages are acquired (902). The method also includes determining aprimary image of the plurality of images, the primary image determinedbased on a portion of the image data from the primary image that isdisplayed on the display monitor compared to portions of the image datafrom other images that are displayed on the display monitor (904). Themethod also includes displaying the image data from the plurality ofimages on the display monitor, the image data from the plurality ofimages displayed from a perspective center of the primary image (906).The method also includes arranging a stacking order of the plurality ofimages so that the primary image is on top (908).

It should be appreciated that the specific steps illustrated in FIG. 9provide a particular method for displaying image data from a pluralityof images according to an embodiment of the present invention. Othersequences of steps may also be performed according to alternativeembodiments. For example, alternative embodiments may perform the stepsoutlined above in a different order. Moreover, the individual stepsillustrated in FIG. 9 may include multiple sub-steps that may beperformed in various sequences. Furthermore, additional steps may beadded or removed depending on the particular application.

Pixel Selection

FIG. 10 is a simplified diagram showing pixels of a display monitor andcorresponding pixels of image data on an image sensor in accordance withan embodiment of the invention. In some embodiments it may be desirableto accurately select a point of interest in an image displayed on adisplay monitor. While image data displayed in accordance with some ofthe embodiments described above has a high metric value, smalldistortions may still exist in the displayed image data. For example, agraphics system typically introduces small distortions in transformingan array of image data to fit onto a display monitor. As a result, smalldifferences may exist between a location of a point of interest asdisplayed on a display monitor and an actual location of the point inthe real world as captured by the image sensor. To eliminate these smalldifferences, some embodiments provide a means by which a point ofinterest can be selected based on the image data as captured by theimage sensor.

This is illustrated with reference to FIG. 10, which shows a continuousscene of image data displayed on a display monitor 1040. A portion 1042of the scene surrounding the point of interest is selected on thedisplay monitor 1040. The portion 1042 of the scene may be selected oridentified using any of a variety of techniques. For example, an areasurrounding the point of interest may be selected using a drag box,selecting corner points, or the like.

Once an area has been selected, pixels of the display monitor alongedges or corners of the area are identified along with theircorresponding pixels of image data. One or more inverse transforms areperformed to transform the pixels of image data to pixels of image dataas captured by an image sensor. This is illustrated in FIG. 10 by dottedlines extending from corners of the portion 1042 of the scenesurrounding the point of interest. The dotted lines extend to aconcave-shaped surface 1028 (representing the transform applied to theimage data using the nonlinear corrections as described above withregard to FIGS. 4A-4B, FIGS. 5A-5B, and FIG. 6), and from theconcave-shaped surface 1028 to pixels 1046 of an image sensor 1044. Inreality the dotted lines are not reflected back to the image sensor butinstead provide a visual representation as the pixels of the displaymonitor are used to identify pixels of image data as captured by animage sensor 1044.

While small distortions may exist between the displayed image data andthe image data as captured by the image sensor 1044, it is highly likelythat the pixels identified on the image sensor 1044 will surround pixelsrepresenting the point of interest. Based on positions of the pixels ofimage data identified on the image sensor 1044, an area of the imagesensor 1044 can be determined and the corresponding image data ascaptured by the image sensor 1044 can be displayed (withouttransformation) in a window (e.g., a pop up window) on the displaymonitor 1040. A user can then select the point of interest using theimage data as captured by the image sensor 1044. In some embodiments theimage data as captured by the image sensor 1044 can be magnified toprovide further ease in identifying the point of interest.

Similar methods may be used in accordance with other embodiments. Forexample, in one embodiment the image data displayed on the displaymonitor 1040 may be downsampled to provide a low resolution image on thedisplay monitor. It may not be possible to accurately identify a pointof interest on the low resolution image. By selecting an areasurrounding the point of interest, however, a corresponding area of theimage data as captured by the image sensor 1044 can be identified andprovided to a user in a window on the display monitor. The image datapresented in the window may be provided in full resolution to allowaccurate identification of the pixels associated with the point ofinterest.

FIG. 11 is a flowchart illustrating a method for identifying one or morepixels of image data associated with a point in an image using a displaymonitor displaying the image in accordance with an embodiment of theinvention. The method includes receiving an indication of a portion ofthe image as displayed on the display monitor that surrounds the point(1102). The method also includes identifying pixels of image data asdisplayed on the display monitor that are located approximately at edgesof the portion of the image (1104). The method also includes performingone or more inverse transformations to transform the pixels of imagedata as displayed on the display monitor to pixels of image data ascaptured by an image sensor (1106). The method also includes identifyinga portion of image data as captured by the image sensor that correspondsto a portion of the image that surrounds the point, the portion of theimage data identified based on the pixels of image data as captured bythe image sensor (1108). The method also includes displaying the portionof the image data as captured by the image sensor on the display monitorwithout data transformation (1110). The method also includes receivingan indication of the one or more pixels of image data associated withthe point in the image (1112).

It should be appreciated that the specific steps illustrated in FIG. 11provide a particular method for identifying pixels of image dataaccording to an embodiment of the present invention. Other sequences ofsteps may also be performed according to alternative embodiments. Forexample, alternative embodiments may perform the steps outlined above ina different order. Moreover, the individual steps illustrated in FIG. 11may include multiple sub-steps that may be performed in varioussequences. Furthermore, additional steps may be added or removeddepending on the particular application.

Identifying a Point of Interest from Different Stations

FIGS. 12A-12B are simplified diagrams showing imaging devices atdifferent stations around a site and a point of interest that iscaptured in images from each of the stations in accordance with anembodiment of the invention. FIG. 12A shows a site 1200 that may be, forexample, a construction site or a room. The outer line surrounding thesite 1200 may correspond to a fence surrounding the construction site orphysical walls bounding the room. Imaging devices 1205, 1215, 1225 areeach at different locations (or stations) around the site 1200. Eachimaging device 1205, 1215, 1225 includes a plurality of imaging devicesarranged in a circle similar to the example shown in FIG. 3 anddescribed above. Lines indicating a field of view of each imaging devicewithin the site 1200 are shown. As can be seen, the entire site iswithin a field of view of at least one (and typically many) of theimaging devices.

FIG. 12A also shows a point of interest 1235 located in an upper cornerof the site 1300, and a shaded region 1230 representing image data thatforms a continuous display of a scene displayed on a display monitor orwithin a window on a display screen. The scene being displayed in thisexample is a portion of the site 1200. It should be appreciated that thescene may extend beyond the boundaries of the site 1200 and thus beyondthe shaded region 1230 indicated in this example. The image data beingdisplayed is captured using the imaging device 1205 (or imaging devicesforming imaging device 1205). As shown by the shaded region 1230, theimage data being displayed includes portions of image data from severalof the imaging devices of the imaging device 1205. The image data isdisplayed from a perspective center located at the same location as oneof the imaging devices, although this is not required. The image datamay be displayed from a perspective center at any location.

It should be appreciated that views of other portions of the site 1200that are obtained from the imaging devices 1215, 1225 located atdifferent stations may also be displayed on the display monitor (e.g.,in different windows). This provides a different view of the site 1200from each of the different stations (or each of the imaging devices1205, 1215, 1225). Corresponding shaded regions are not shown in FIG.12A merely for ease of illustration.

The point of interest 1235 may be identified in any of the views fromany of the imaging devices 1205, 1215, 1225 displayed on the displaymonitor. In this particular example, the point of interest 1235 isidentified in the region 1230 that is displayed from imaging device1205. In some situations, it may be desirable to locate the same pointof interest 1235 in views from each of the other imaging devices 1215,1225 as well (e.g., for position measurement using triangulationmethods). Because each imaging device captures a 360° view of the site1200, however, it may be time consuming to pan around until the point ofinterest 1235 is identified in views from each of the imaging devices.Additionally, the point of interest 1235 may simply be a point on theground with no identifying features. In this case it may be difficult(if not impossible) to accurately select the point of interest 1235 ineach of the views.

In accordance with an embodiment, a reference graphic (e.g., a line) maybe superimposed over the point of interest 1235 to aid in identifying itin each of the views. Each of the views may be automatically oriented onthe display monitor to show the reference graphic. As an example, inFIG. 12B a line 1245 is used as a reference graphic. A position andorientation of the line 1245 in the view represented by the shadedregion 1230 may be determined by identifying pixels on an image sensorof the imaging device 1205 that correspond to the image datarepresenting the point of interest 1235. A ray may be mathematicallydetermined that extends from the pixels and passes through the point ofinterest 1235. The line 1245 may extend along this ray and besuperimposed over the image data displayed on the display monitor.

The line 1245 may not be visible in the views on the display monitorfrom the imaging devices 1215, 1225, but once the position andorientation of the line 1245 is determined the views could bere-oriented if necessary such that the line 1245 is visible in each ofthe views. The line 1245 can be followed in each of the views and thepoint of interest 1235 can be located along the line 1245.

Some embodiments may also include a slidable indicator 1240 to aide inidentifying the point of interest. The slidable indicator 1240 may beanother reference graphic that is configured to be moved along the line1245 with user input. In an embodiment, the slidable indicator 1240 maybe movable from any of the views displayed on the display monitor (notjust the view from imaging device 1205). In this case, the slidableindicator 1240 may be moved to the point of interest in whichever viewthe point of interest 1235 is most easily identified. In someembodiments, as the slidable indicator 1240 is moved along the line 1245in one view, the other views displayed on the display monitor areautomatically adjusted such that each of the views follow the movementof the slidable indicator 1240 along the line 1245. Further, each of theviews may be automatically adjusted to a common zoom level that aides infollowing the slidable indicator.

FIG. 13 is a flowchart illustrating a method for locating a point ofinterest in a plurality of images in accordance with an embodiment ofthe invention. The method includes identifying the point of interest ina first view of the scene, the point of interest identified by selectinga point on the display monitor that corresponds to the point of interestas displayed in the first view (1302). The method also includesidentifying pixels of image data that correspond to the selected pointon the display monitor, the pixels of image data being part of one ormore arrays of image data that form the first view of the scene (1304).The method also includes determining a line in each of the views, theline extending along a ray that passes through the point of interest inthe scene to pixels of an image sensor corresponding to the pixels ofimage data in the first view of the scene (1306). The method alsoincludes displaying the line in each of the views in which the line isvisible (1308). The method also includes providing a slidable indicatorin each of the views that is movable along the line, where the slidableindicator can be moved by user input, and any movement of the slidableindicator along the line is visible in each of the other views andupdates the scene displayed in each of the other views accordingly(1310). The method also includes moving the slidable indicator along theline until the slidable indicator is proximate the point of interest ineach of the other views (1312).

It should be appreciated that the specific steps illustrated in FIG. 13provide a particular method for locating a point of interest in aplurality of images according to an embodiment of the present invention.Other sequences of steps may also be performed according to alternativeembodiments. For example, alternative embodiments may perform the stepsoutlined above in a different order. Moreover, the individual stepsillustrated in FIG. 13 may include multiple sub-steps that may beperformed in various sequences. Furthermore, additional steps may beadded or removed depending on the particular application.

It should be appreciated that some embodiments of the present inventionmay be implemented by hardware, software, firmware, middleware,microcode, hardware description languages, or any combination thereof.When implemented in software, firmware, middleware, or microcode, theprogram code or code segments to perform the necessary tasks may bestored in a computer-readable medium such as a storage medium.Processors may be adapted to perform the necessary tasks. The term“computer-readable medium” includes, but is not limited to, portable orfixed storage devices, optical storage devices, wireless channels, simcards, other smart cards, and various other non-transitory mediumscapable of storing, containing, or carrying instructions or data.

While the present invention has been described in terms of specificembodiments, it should be apparent to those skilled in the art that thescope of the present invention is not limited to the embodimentsdescribed herein. For example, features of one or more embodiments ofthe invention may be combined with one or more features of otherembodiments without departing from the scope of the invention. Thespecification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense. Thus, the scope of thepresent invention should be determined not with reference to the abovedescription, but should be determined with reference to the appendedclaims along with their full scope of equivalents.

What is claimed is:
 1. A method for displaying image data from aplurality of images on a display monitor, where at least some of theplurality of images are acquired from a plurality of imaging devices,the method comprising: displaying the image data from the plurality ofimages on the display monitor, the image data from the plurality ofimages displaying an image perceived from a perspective centerassociated with different locations, wherein each image of the pluralityof images are captured by the plurality of imaging devices pointedradially outward from the perspective center, and wherein each imagingdevice of the plurality of imaging devices is in a fixed relationship toeach other with a fixed angle between them, and wherein the displayedimage comprises less than all images of the plurality of images capturedby the plurality of imaging devices pointed radially outward from theperspective center; determining a primary image of the plurality ofimages, the primary image determined based on a portion of the imagedata from the primary image that is displayed on the display monitorcompared to portions of the image data from other images that aredisplayed on the display monitor; thereafter displaying the image datafrom the plurality of images on the display monitor, the image data fromthe plurality of images displayed from a perspective center of theprimary image; and arranging a stacking order of the plurality of imagesso that the primary image is on top.
 2. The method of claim 1 whereinthe plurality of images capture overlapping portions of a scene.
 3. Themethod of claim 1 wherein the plurality of images capture adjacentportions of a scene.
 4. The method of claim 1 wherein the perspectivecenter associated with the different positions from which the pluralityof images are acquired is located substantially at an average positionof the different positions.
 5. The method of claim 1 wherein the primaryimage is also determined based on a position of the primary image on thedisplay monitor relative to positions of the other images displayed onthe display monitor.
 6. The method of claim 1 further comprising forminga line around the primary image as an indication of the primary image.7. A display apparatus, comprising: a display monitor configured todisplay image data from a plurality of images, where at least some ofthe plurality of images are acquired from a plurality of imagingdevices; one or more processors configured to: prior to a time T1,arrange the image data from the plurality of images for display on thedisplay monitor, so that the image data from the plurality of images candisplay an image perceived from a perspective center associated withdifferent locations, wherein each image of the plurality of images arecaptured by the plurality of imaging devices pointed radially outwardfrom the perspective center, and wherein each imaging device of theplurality of imaging devices is in a fixed relationship to each otherwith a fixed angle between them, and wherein the displayed imagecomprises less than all images of the plurality of images captured bythe plurality of imaging devices pointed radially outward from theperspective center; and determine a primary image of the plurality ofimages, the primary image determined based on a portion of the imagedata from the primary image that is displayed on the display monitorcompared to portions of the image data from other images that aredisplayed on the display monitor; after the time T1, arrange the imagedata from the plurality of images for display on the display monitor, sothat the image data from the plurality of images can be displayed from aperspective center of the primary image; and arrange a stacking order ofthe plurality of images so that the primary image is on top.
 8. Thedisplay apparatus of claim 7 wherein the plurality of images captureoverlapping portions of a scene.
 9. The display apparatus of claim 7wherein the plurality of images capture adjacent portions of a scene.10. The display apparatus of claim 7 wherein the perspective centerassociated with the different positions from which the plurality ofimages are acquired is located substantially at an average position ofthe different positions.
 11. The display apparatus of claim 7 whereinthe primary image is also determined based on a position of the primaryimage on the display monitor relative to positions of the other imagesdisplayed on the display monitor.
 12. The display apparatus of claim 7wherein the processor is further configured to form a line around theprimary image as an indication of the primary image.